
United States Patent and Trademark Office 




UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 

Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 22313-1450 
www.uspto.gov 



APPLICATION NO. 


FILING DATE 


FIRST NAMED INVENTOR 


ATTORNEY DOCKET NO. 


CONFIRMATION NO. 


09/925,397 


08/09/2001 


Yuan-Chi Chang 


YOR9-200 1-0287 


4473 








(8728-514) 





46069 7590 12/27/2007 

F. CHAU & ASSOCIATES, LLC 
130 WOODBURY ROAD 
WOODBURY, NY 11797 



EXAMINER 



EHICHIOYA, FRED I 



ART UNIT 



2162 



PAPER NUMBER 



MAIL DATE 



12/27/2007 



DELIVERY MODE 



PAPER 



Please find below and/or attached an Office communication concerning this application or proceeding. 

The time period for reply, if any, is set in the attached communication. 



PTOL-90A (Rev. 04/07) 



• 


Application No. 


Applicant(s) 


Office Action Summary 


09/925,397 


CHANG ETAL. 


Examiner 


Art Unit 






Fred 1. Ehichioya 


2162 





The MAILING DATE of this communication appears on the cover sheet with the correspondence address » 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .1 36(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1)[EI Responsive to communication(s) filed on 04 October 2007 . 
2a)D This action is FINAL. 2b)KI This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) E3 Claim(s) 20-39 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) E0 Claim(s) 20-39 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) ; are subject to restriction and/or election requirement. 

Application Papers 

9) Q The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1 .121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 
a)D All b)D Some * c)Q None of: 

1 .□ Certified copies of the priority documents have been received. 

2. Q Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attachment(s) 

1) ^ Notice of References Cited (PTO-892) 4) J^J Interview Summary (PTO-413) 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) Pa P er No(s)/Mail Date. . 

3) □ Information Disclosure Statement(s) (PTO/SB/08) 5 > □ Notice of lnformal Patent Application 

Paper No(s)/Mail Date . 6) U Other: . 



U.S. Patent and Trademark Office 
PTOL-326 (Rev. 08-06) 



Office Action Summary 



Part of Paper No./Mail Date 20071217 



Application/Control Number: 09/925,397 Page 2 

Art Unit: 2162 

DETAILED ACTION 
Continued Examination Under 37 CFR 1.114 

1 . A request for continued examination under 37 CFR 1.114, including the fee set 
forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this 
application is eligible for continued examination under 37 CFR 1.114, and the fee set 
forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action 
has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on October 
4, 2007 has been entered. 

2. claims 20 - 39 are pending in this Office Action 

Response to Arguments 

3. Applicant's arguments with respect to claims 20, 27 and 34 have been 
considered but are moot in view of the new ground(s) of rejection. 

Claim Rejections - 35 USC § 103 

4. The following is a quotation of 35 U.S. C. 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

This application currently names joint inventors. In considering patentability of 
the claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of 
the various claims was commonly owned at the time any inventions covered therein 
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were made absent any evidence to the contrary. Applicant is advised of the obligation 
under 37 CFR 1.56 to point out the inventor and invention dates of each claim that was 
not commonly owned at the time a later invention was made in order for the examiner to 
consider the applicability of 35 U.S.C. 103(c) and potential 35 U.S.C. 102(e), (f) or (g) 
prior art under 35 U.S.C. 103(a). 

5. Claims 20, 27 and 34 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Non-Patent Document: "Relevance Feedback: A Power Tool for Interactive 
Content-Based Image Retrieval" by Rui et al (hereinafter "Rui") in view of Non-Patent 
Document: "A Fuzzy Object Query Language (FOQL) for Image databases" by Nepal et 
al (hereinafter "Nepal"). 

Regarding claims 20 and 27, Rui teaches a method for processing multimedia 
data in a computer system, comprising: 

receiving as input a high-level concept describing data to be accessed (page 
644, Abstract: "During the retrieval process, the user's high-level query and perception 
subjectivity are captured"); 

translating the high-level concept into a low-level query by using stored concept 
constructs which are defined using features derived from a plurality of application 
domains (page 645, column 1, paragraph 1: "mapping a high-level concept (fresh apple) 
to low-level features (color or shape)" - the domains are fruit, color and shape and also 
page 645, column 2, paragraph 1: "high-level concepts can be captured by low-level 
features"). 
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Rui does not explicitly teach transferring the low-level query to one or more 
search engines as claimed. 

Nepal teaches transferring the low-level query to one or more search engines to 
access information using the low-level query (page 123, column 2, paragraph 1: "the 
image retrieval is accomplished based on low level features such as color and texture"). 

It would have been obvious to one of ordinary skill in the data processing art at 
the time of the present invention to combine teaching of the cited references because 
Nepal's teaching of "transferring the low-level query to one or more search engines to 
access information using the low-level query" would have allowed Rui's system to 
formulate queries and handle problems such as mapping low level features to high level 
concepts. The motivation is that these queries and data in image databases are 
inherent in nature. 

Regarding claim 34, Rui discloses a system for processing multimedia data in a 
computer system, comprises: 

a concept translation engine that receives a high-level concept describing data to 
be accessed page 644, Abstract: "During the retrieval process, the user's high-level 
query and perception subjectivity are captured 1 '), translates the high-level concept into a 
low-level query using a hierarchy of stored concept constructs which are defined by 
features derived from a plurality of application domains (page 645, column 1, paragraph 
1: "mapping a high-level concept (fresh apple) to low-level features (color or shape)" - 
the domains are fruit, color and shape and also page 645, column 2, paragraph 1 : "high- 
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level concepts can be captured by low-level features"; It is implicit and inherent in this 
situation that there is a type of code, algorithm or engine that maps/translates high-level 
concept to low-level concept). 

Rui does not explicitly teach constraint or transferring the low-level query to one 
or more search engines as claimed. 

Nepal discloses constraints (page 121 , column 2: "constraint specified") among 
sibling elements in the hierarchy (page 120, column 1: "a collection of images can be 
classified into various groups ... the inverse relationship of the images with their 
respective collection"). 

and transfers the low-level query to one or more search engines to access 
information using the low-level query (page 123, column 2, paragraph 1: "the image 
retrieval is accomplished based on low level features such as color and texture"); and 

a concept repository for storing and accessing the concept constructs (page 123, 
column 2, paragraph 2: "concept definitions are stored in the database"). 

It would have been obvious to one of ordinary skill in the data processing art at 
the time of the present invention to combine teaching of the cited references because 
Nepal's teaching of "transferring the low-level query to one or more search engines" 
would have allowed Rui's system to formulate queries and handle problems such as 
mapping low level features to high level concepts. The motivation is that these queries 
and data in image databases are inherent in nature. 
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6. Claims 21 - 26, 28 - 33, and 35 - 39 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Rui in view of Nepal and further in view of Non-Patent 
Document: "CAMEL: Concept Annotated iMagE Libraries", In Storage and Retrieval for 
Image and Video Database, San Jose, CA, Jan. 2001. SPIE (numbered pages 1 - 12), 
Author: Natsev et al (hereinafter "Natsev"). 

Regarding claims 21 , 28 and 35, Rui and Nepal disclose the claimed subject 
matter as discussed in claims 20, 27 and 34 respectively. 

Rui or Nepal does not disclose concept library as claimed. 

Natsev discloses storing the concept constructs in a concept library module (see 
Fig. 1 "concept library" and Section 1.2 "The concept library is module for a persistent 
storage of concepts"); 

storing the features in a feature library module (see Fig. 3 "Feature index"); 

storing constraints in a constraint library module (see section 6, "Another 
improvement that we are considering is the introduction of spatial constraints in the 
query engine. For example ... the corresponding document is ranked higher"); and 

storing matching algorithms in a matching algorithm library module (see section 
3: "Since the feature. . . , the region matching definition still applies" - "matching 
definition" is "matching algorithm"). 

It would have been obvious to one of ordinary skill in the data processing art at 
the time of the present invention to combine teaching of the cited references because 
Natsev's teaching of storing the concept constructs in a concept library module, storing 
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the features in a feature library module and storing constraints in a constraint library 
module would have presented a new direction for improving current image query 
system, namely focusing the query specification part as gateway to better performance 
and usability (see Natsev section 6) for the combination Rui and Nepal's system. 

Regarding claims 22, 29 and 36; Natsev teaches interfacing the library modules 
to the application domains (see fig.3 and section 4, "Fig.3 illustrates the architecture of 
the ... , as well as the Concept Library module"). 

Regarding claims 23, 30 and 37, Natsev teaches building a concept construct 
(see fig. 1 and section 1.2, The concept cataloguing, or learning, phase is used to define 
visual concepts and build concept library. Concepts are defined . . . and to associate it 
with the given concept"). 

Regarding claims 24 and 31 , Natsev teaches a method as defined in Claim 23, 
wherein the step of building a concept construct comprise combining one or more of the 
features with see section 1.2, The concept cataloguing, or learning, phase is used to 
define visual concepts and build concept library) 

zero or more of the stored concept (see Fig. 1 and section 1 .2, "The concept 
library is a module for persistent storage of concept"), and 

zero or more of the constraints (see section 6, "Another improvement that we are 
considering is the introduction of spatial constraint in query engine"). 
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Regarding claims 25 and 32, Nepal teaches a concept construct is represented 
using a hierarchical fuzzy graph data tree-structure comprising: 

nodes that correspond to child-concepts and a subset of the features (page 120, 
column 1: "a collection of images can be classified into various groups ... the inverse 
relationship of the images with their respective collection"); 

aggregation edges that correspond to parent-child relationships (page 119, 
section 3: "The Img-comp type is defined for image component objects, the Image type 
for image objects and Img-col type for collection of images. In the figure, the relationship 
types are shown by lines"); and 

association edges that correspond to inter-sibling constraints (see page 119, 
column 1 : A fuzzy set of object is a (usual) set of objects, with each object having 
associated membership value"). 

Regarding claims 26 and 33, Natsev teaches a method as defined in Claim 20, 
wherein the features are user defined (see section 2, "One of the most commonly used 
image features is color histogram, . . . places the burden on the user to specify weights 
for the different features"). 

Regarding claim 38, Natsev teaches a system as defined in Claim 34, wherein 
the translation engine further comprises an interpreter that translates the high level 
concept (see fig. 1 ). 
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Regarding claim 39, Natsev teaches a system as defined in Claim 34, further 
comprises a search engine (see section 2, "the high-level approach that . . . . , there are 
some considerations that make certain query engines more suitable than others". 



7. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Fred I. Ehichioya whose telephone number is 571-272- 
4034. The examiner can normally be reached on M - F 8:00 AM to 4:30 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, John E. Breene can be reached on 571-272-4107. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 



Conclusion 



/Fred I. Ehichioya/ 




